Deep equilibrium networks are sensitive to initialization statistics

Deep equilibrium networks (DEQs) are a promising way to construct models which trade off memory for compute. However, theoretical understanding of these models is still lacking compared to traditional networks, in part because of the repeated application of a single set of weights. We show that DEQs are sensitive to the higher order statistics of the matrix families from which they are initialized. In particular, initializing with orthogonal or symmetric matrices allows for greater stability in training. This gives us a practical prescription for initializations which allow for training with a broader range of initial weight scales

Understanding plastic deformation in thermal glasses from single-soft-spot dynamics

By considering the low-frequency vibrational modes of amorphous solids, Manning and Liu [Phys. Rev. Lett. 107, 108302 (2011)] showed that a population of "soft spots" can be identified that are intimately related to plasticity at zero temperature under quasistatic shear. In this work we track individual soft spots with time in a two-dimensional sheared thermal Lennard Jones glass at temperatures ranging from deep in the glassy regime to above the glass transition temperature. We show that the lifetimes of individual soft spots are correlated with the timescale for structural relaxation. We additionally calculate the number of rearrangements required to destroy soft spots, and show that most soft spots can survive many rearrangements. Finally, we show that soft spots are robust predictors of rearrangements at temperatures well into the super-cooled regime. Altogether, these results pave the way for mesoscopic theories of plasticity of amorphous solids based on dynamical behavior of individual soft spots.Comment: 9 pages, 6 figure